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FOREWORD 


The  Human  Factors  Technical  Area  of  the  Army  Research  Institute  (ARI)  is 
concerned  with  the  demands  of  increasingly  complex  battlefield  systems  that  are 
used  to  acquire,  transmit,  process,  disseminate,  and  utilize  information.  This 
increased  complexity  places  greater  demands  upon  the  operator  interacting  with 
the  machine  system.  Research  in  this  area  is  focused  on  human  performance 
problems  related  to  interactions  within  command  and  control  centers  as  well  as 
on  issues  of  systems  development.  Such  research  is  concerned  with  software  de¬ 
velopment,  topographic  products  and  procedures,  tactical  symbology,  user- 
oriented  systems,  information  management,  staff  operations  and  procedures,  de¬ 
cision  support,  and  sensor  systems  integration  and  utilization. 

An  issue  of  special  concern  within  the  area  of  user-oriented  systems  is 
the  improvement  of  manual  data  input  procedures,  especially  in  battlefield  au¬ 
tomated  systems.  The  main  source  of  information  for  tactical  data  systems  is 
manual  data  entry — a  slow,  difficult,  and  error-prone  process.  The  capability 
of  tactical  data  systems  such  as  the  Tactical  Operating  System  (TOS)  to  support 
command  staff  actions  with  accurate,  complete,  and  timely  information  is  depen¬ 
dent  on  the  performance  of  the  person  who  must  manually  enter  information  into 
the  system.  ARI ' s  research  program  on  data  entry  has  resulted  in  simplified 
message  formats,  improved  reference  codes,  aids  for  on-line  preparation  and 
verification  of  message  entries,  improved  training  procedures,  techniques  for 
error  analysis,  and  simulation  techniques  to  aid  in  the  design  process.  The 
present  publication  summarizes  the  research  performed  on  the  data  entry  process 
by  ARI  over  the  past  decade.  The  rationale  of  ARI's  approach  to  the  problems, 
the  findings,  operational  implications,  and  further  research  requirements  are 
presented. 

Research  in  user-oriented  systems  is  conducted  as  an  in-house  effort  aug¬ 
mented  through  contracts.  This  report  resulted  from  an  in-house  research  ef¬ 
fort  responsive  to  requirements  of  Army  Project  2Q162717A790  and  to  special 
requirements  of  the  U.S.  Army  Combined  Arms  Combat  Development  Activity,  Fort 
Leavenworth,  Kansas.  Special  requirements  are  contained  in  Human  Resource  Need 
80-304,  "Interactive  Procedures  for  Data  Inputting,  Organization,  Retrieval  and 
Purge. " 
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RECENT  ARI  RESEARCH  ON  THE  DATA  ENTRY  PROCESS 
IN  BATTLEFIELD  AUTOMATED  SYSTEMS 


BRIEF 


Requirements : 

To  increase  the  efficiency  and  accuracy  with  which  operators  enter  data 
into  battlefield  automated  systems. 


Procedure: 

This  paper  reviews  ARI  research  designed  to  improve  the  data  entry  proc¬ 
ess.  The  first  and  second  sections  of  the  paper  describe  the  data  entry  proc¬ 
ess  in  general  as  well  as  in  the  context  of  a  specific  battlefield  automated 
system,  the  Tactical  Operating  System  (TOS) .  Because  it  was  used  as  an  exem¬ 
plar  of  the  data  entry  process,  TOS  played  an  important  role  in  the  development 
of  improved  data  entry  procedures.  The  third  section  of  the  paper  reviews  the 
findings  and  conclusions  of  the  many  ARI  research  projects  concerned  with  data 
entry .  Among  the  areas  covered  in  ARI 1 s  research  program  are : 

a.  How  to  format  and  display  data  entry  information. 

b.  What  safeguards  can  be  developed  to  reduce  the  number  of  operator 
errors  made  and/or  accepted  by  the  system. 

c.  What  kinds  of  operator  job  aids  can  be  developed  to  improve 
performance. 

d.  How  to  improve  operator  training. 

e.  How  to  make  the  system's  message  codes  easier  to  use  and  more 
memorable . 

f.  How  to  improve  the  design  of  keyboards. 

The  fourth  section  of  the  paper  reports  on  efforts  to  analyze  the  causes  of  op¬ 
erator  errors.  This  section  also  discusses  the  development  of  a  simulation  of 
the  data  entry  process.  The  simulation  is  intended  to  facilitate  system  design 
by  permitting  the  inexpensive  evaluation  of  alternative  data  entry  procedures. 
The  fifth  section  presents  a  general  discussion  of  the  problems  that  have  been 
encountered  by  the  ARI  research  program.  Also  included  here  is  a  discussion  on 
how  this  program  might  be  improved  in  the  future.  The  final  section  of  the 
paper  summarizes  the  operational  implications  of  ARI ' s  research  results. 
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Findings: 
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ARI  research  has  contributed  to  the  improvement  of  the  data  entry  process 
in  battlefield  automated  systems.  Major  findings  and  operational  implications 
derived  from  the  ARI  program  include: 

a.  The  number  of  different  formats  used  in  data  entry  should  be  reduced 
through  possible  consolidation. 

b.  The  transformation  and  input  of  the  TOS  message  should  be  performed 
by  a  single  operator  and  as  a  single  operation. 

c.  Menu  selection  is  the  superior  technique  for  data  entry. 

d.  Message  formats  are  best  designated  by  the  use  of  an  all  letter  code 
as  opposed  to  a  mixed  alpha-numeric  code. 

e.  Flexibility  and  cost  efficiency  in  the  design  of  data  entry  systems 
can  be  achieved  through  improved  computerized  modeling. 

f.  Response-sensitive  training  is  an  effective  method  for  reducing  train¬ 
ing  time  without  sacrificing  input  accuracy. 

g. ^  The  Alpha-dot  keyboard  has  value  as  a  data  entry  device  for  use  by 

frontline  observers  on  the  battlefield. 

h.  Approximately  80%  of  all  data  entry  errors  are  not  detectable  by  com¬ 
puterized  error  detection  routines. 


Utilization  and  Findings: 

This  report  brings  together  the  principal  results  of  ARI's  research  efforts 
in  the  area  of  data  entry.  It  provides  interested  system  proponents  and  devel¬ 
opers  with  convenient  access  to  recommendations  and  guidance  for  improving  the 
data  entry  process  in  battlefield  automated  systems. 
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RECENT  ARI  RESEARCH  ON  THE  DATA  ENTRY  PROCESS 
IN  BATTLEFIELD  AUTOMATED  SYSTEMS 


INTRODUCTION 

The  U.S.  Army  is  engaged  in  the  development  of  over  90  different  tactical 
data  systems  which  will  improve  the  effectiveness  and  efficiency  of  information 
processing  on  the  battlefield.  By  automating  many  of  the  clerical  and  manual 
processing  tasks  involved  in  tactical  information  processing,  these  new  systems 
will  ameliorate  the  burdensome,  time  consuming,  and  error  prone  chores  related 
to  the  collection,  processing,  and  presentation  of  information.  These  systems 
will  provide  field  users  with  improved  access  to  a  more  accurate  and  a  more 
comprehensive  data  base  of  tactical  information. 

The  data  entry  process  is  critical  to  the  operation  and  maintenance  of  any 
battlefield  automated  system.  Delays  and  errors  that  occur  during  data  entry 
deprive  system  users  of  timely  and  accurate  information.  Battlefield  automated 
systems  can  be  no  better  than  the  quality  of  the  data  entered  into  them.  When 
the  data  input  process  is  degraded,  the  worth  of  the  total  system  is  compro¬ 
mised.  Although  elements  of  the  data  entry  process  will  be  automated,  much  of 
it  will  depend  upon  human  performance.  For  the  system  to  be  successful,  both 
the  hardware  and  its  human  operators  must  be  capable  of  operating  in  a  combat 
environment  where  they  are  exposed  to  performance-degrading  conditions. 

This  paper  reviews  ARI  research  designed  to  improve  the  data  entry  proc¬ 
ess.  (An  earlier  presentation  of  this  material  was  made  by  Alderman,  1976.) 

The  next  section  of  the  paper  describes  the  data  entry  process  in  genera]  as 
well  as  in  the  context  of  a  specific  battlefield  automated  system,  the  Tactical 
Operating  System  (TOS) .  Because  it  was  used  as  an  example  of  the  data  entry 
process,  TOS  played  an  important  role  in  the  development  of  improved  data  entry 
procedures.  The  third  section  of  the  paper  reviews  the  findings  and  conclusions 
of  the  many  ARI  research  projects  concerned  with  data  entry.  Among  the  areas 
covered  in  ARI ' s  research  program  are: 

•  How  to  format  and  display  data  entry  information. 

•  What  safeguards  can  be  developed  to  reduce  the  number  of  operator  errors 
made  and/or  accepted  by  the  system. 

•  What  kinds  of  operator  job  aids  can  be  developed  to  improve  performance. 

•  How  to  improve  operator  training. 

•  How  to  make  the  system's  message  codes  easier  to  use  and  more  memorable. 

•  How  to  improve  the  design  of  keyboards. 

The  fourth  section  of  the  paper  reports  on  efforts  to  analyze  the  causes  of  op¬ 
erator  errors.  This  section  also  discusses  the  development  ol  a  simulation  of 
the  data  entry  process.  The  simulation  is  intended  to  facilitate  system  design 
by  permitting  the  inexpensive  evaluation  of  alternate  data  entry  procedures. 

The  fifth  section  presents  a  general  discussion  of  the  problems  that  have  been 
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encountered  by  the  AR1  research  program.  Also  included  here  is  a  discussion  on 
how  this  program  might  be  improved  in  the  future.  The  final  section  of  the 
paper  summarizes  the  operational  implications  of  API's  research  results. 

ThE  DATA  ENTRY  PROCESS  IN  BATTLEFIELD  AUTOMATED  SYSTEMS 
Outline  of  the  Data  Entry  Process 

Five  critical  operations  must  b.  performed  during  the  course  of  a  user- 
computer  interaction  (Ringel,  Vicino,  &  Andrews,  196G) .  Figure  1  provides  a 
graphic  representation  of  these  operations  and  of  the  information  flow  associ¬ 
ated  with  them.  Three  of  these  five  operations  constitute  the  data  entry 
process : 

Screen  incoming  data  for  pertinence,  credibility,  impact,  priority,  and 
routing. 

Transform  the  raw  data  for  input  into  automated  storage  devices. 

Input  the  transformed  data  into  storage  devices  for  subsequent  computation 
and  display. 

The  two  remaining  operations  are  assimilation  of  the  displayed  data  and  deciding 
on  courses  of  action.  These  operations  are  not  part  of  the  data  entry  process 
per  se  and  are  therefore  not  discussed  in  this  paper. 

The  data  entry  process  begins  with  the  reception  of  messages  or  data  for 
inclusion  in  the  data  base.  These  messages  can  vary  widely  in  terms  of  their 
relevance,  content,  form,  and  completeness.  Therefore,  the  first  step  in  the 
data  entry  process  is  to  screen  the  incoming  messages  and  exclude  from  data 
entry  those  that  are  irrelevant  (see  Figure  2) .  However,  most  experiments  and 
training  exercises  employ  only  relevant  messages  in  their  scenarios.  Ambiguous 
data,  erroneous  data,  deceitful  data,  and  data  out  of  context  are  not  included. 
Thus,  the  screening  operation  tends  to  receive  little  attention  during  most 
test  and  design  stages.  Although  the  screening  operation  determines  the  con¬ 
tent  of  the  data  base  and  may  delay  the  relaying  of  messages,  research  emphasis 
has  typically  been  placed  on  the  transform  operation. 

The  transform  operation  involves  the  selection  of  the  appropriate  format 
for  translating  the  message  data  from  free  text  into  a  style  acceptable  for 
data  entry.  Restrictions  on  the  size  of  the  data  base  often  necessitate  the 
use  of  abbreviations,  codes,  and  other  representations.  A  number  of  formats 
are  available  for  the  transformation  of  incoming  messages.  Each  format  is 
tailored  to  a  different  type  of  message  as  well  as  to  the  data  base  operation 
to  be  performed.  Worksheets  are  usually  used  by  operators  to  assist  them  in 
performing  the  transformation  task. 

The  input  operation  involves  the  physical  process  of  using  an  input  device 
to  transcribe  data  from  messages  or  worksheets  into  the  computer.  Most  systems 
provide  a  standard  keyboard  for  entry  of  the  message  as  well  as  a  cathode  ray 
tube  (CRT)  for  immediate  display  of  the  entered  message.  Operators  verify  the 
correctness  of  the  transcribed  message  against  the  worksheet  prior  to  entering 
it  into  the  data  base. 
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Schematic  representation  of  operations  and  information  flow  in  an  automated  TOS 
(Ringel ,  Vicino,  &  Andrews,  1966) 


TOS 


xOb  was  a  battlefield  automated  system  designed  to  provide  command  staffs 
with  a  computerized  method  for  the  storage  and  retrieval  of  battlefield  informa¬ 
tion.  The  development  of  TOS  in  a  user  setting  was  begun  in  1965  and  resulted 
in  a  Seventh  Army  TOS  (SATOS) ,  designed  from  commercial  equipment,  being  intro¬ 
duced  into  the  field  for  testing.  Upon  completion  of  testing  in  about  1969, 
Headquarters,  Modern  Army  Selected  System  Test,  Evaluation  and  Review  (MASSTER) 
relocated  the  experimental  TOS  to  Fort  Hood,  Texas,  in  order  to  support  the  De¬ 
velopmental  TOS  (DEVTOS)  activity.  This  in  turn  led  to  an  effort  to  develop  a 
division  level  system  known  as  the  TOS  Operable  Segment  (TOS2) .  in  1980,  devel¬ 
opment  of  a  second  generation  command  support  system,  SIGMA,  was  initiated.  (For 
further  discussion  of  the  early  history  of  TOS,  see  Baker,  1972a.) 

The  TOS  configuration  (circa  1970)  is  shown  in  Figure  3.  TOS  consisted  of 
the  following  functional  groups: 

•  User  Input/Output  Device  (UIOD) — enables  remote  users  to  communicate 
with  the  system  and  other  users;  consists  of  a  television  type  cathode 
ray  tube  (CRT)  display  with  an  associated  keyboard  and  a  typewriter 
printer . 

•  Remote  Station  Data  Terminal  (RSDT) — an  intermediate  message  processor/ 
transmitter  between  UIODs  and  the  Central  Computing  Center. 

•  Central  Computing  Center  (CCC) --stores  the  TOS  data  base,  performs  TOS 
computations  and  data  manipulations,  and  processes  all  messages  to  and 
from  the  RSDTs . 

Data  entry  in  the  experimental  TOS  begins  with  an  incoming  message  being 
received  from  a  remote  unit.  This  message,  usually  in  the  form  of  free  text, 
is  then  translated  into  the  TOS  message  format  by  an  action  officer.  In  per¬ 
forming  this  task,  the  action  officer  is  guided  by  TOS  message  worksheets. 

Next,  an  operator  transcribes  the  message  from  the  worksheet  to  the  UIOD.  After 
the  transcribed  data  is  verified  against  the  worksheet,  the  operator  transmits 
it  to  the  CCC.  The  input  data  is  then  edited  and  verified  by  the  CCC.  If  any 
errors  are  detected  by  the  CCC,  an  error  message  is  sent  to  the  operator.  When 
this  occurs,  the  operator  corrects  the  error  and  reenters  the  data  into  the 
systems.  When  the  data  is  finally  accepted,  the  CCC  updates  the  data  base. 

The  messages  processed  by  the  action  officer  can  be  categorized  as  either 
operational  or  relay.  Relay  messages  are  free  text  messages  from  one  user  to 
another  and  require  only  retransmission;  thus  they  do  not  affect  the  data  base. 
Operational  messages  are  messages  which  modify  the  data  base  or  which  require  a 
system  response.  They  are  of  the  following  types: 

•  Data  messages — serve  to  enter  data  into  the  system  data  base  and  to 
add,  change,  or  delete  previously  entered  data. 

•  Query  messages--used  to  search  the  data  base  for  specific  information 
which  is  then  output  on  the  typewriter  printer. 

•  Special  Process  Request  (SPR) — used  to  perform  calculations,  summaries, 
and  other  manipulations  of  the  stored  data. 
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•  Standing  Request  for  Information  (SRI) — serves  to  establish  automated 
and  immediate  dissemination  of  pre-specif ied  information  to  designated 
UIODS . 

Entry  of  these  operational  messages  requires  the  completion  of  highly  structured 
worksheets.  A  large  number  of  different  worksheets  are  required  to  accommodate 
all  possible  combinations  of  message  types  and  functional  areas.  Operational 
messages  may  require  more  than  one  worksheet  to  enter  all  the  relevant  data  in 
an  incoming  message  (see  Baker,  1972a) . 

Many  of  the  early  data  entry  studies  performed  by  ARI  used  the  TOS  design 
as  an  exemplar  of  battlefield  automated  systems.  By  so  doing,  these  early  ex- 
pi  '  iments  were  made  more  realistic  and  their  results  were  directly  usable  by 
both  TOS  and  other  prospective  systems. 


RESEARCH  REVIEW 

Research  on  data  entry  has  revolved  around  the  transform  and  the  input  op¬ 
erations.  In  most  instances,  these  operations  were  studied  together  and  there¬ 
fore  they  are  discussed  together  below.  In  seeking  to  improve  the  data  entry 
process,  research  efforts  centered  on  the  following  features:  message  format 
selection,  on-line  versus  off-line  transformation  of  free  text,  verification  by 
the  operator  of  the  input  stage,  job  aids  for  the  operator,  training  procedures, 
typing  aids  and  new  keyboard  designs.  A  summary  of  this  program  is  provided  in 
Table  1. 

•  Format  Selection  is  a  critical  part  of  the  transform  process  during 
data  entry.  For  the  experimental  TOS,  users  are  required  to  select  the  correct 
format (s)  (for  transforming  a  message)  from  about  500  choices.  In  the  majority 
of  cases,  the  transformation  of  a  single  English  text  message  will  require  the 
use  of  more  than  one  format  (Baker,  1972a) .  A  sample  format  is  shown  in  Fig¬ 
ure  4.  Baker,  Mace  and  McKendry  (1969)  investigated  how  to  improve  performance 
during  format  selection.  In  their  experiment,  operators  were  assigned  to  one 
or  two  format  selection  procedures.  One  group  used  a  menu  type  listing  of 
available  TOS  formats.  The  second  group  used  a  simple  job  aid  which  provided 
the  same  information  but  in  the  form  of  a  table  of  reference  codes.  Data  entry 
was  performed  manually  (i.e.,  handwritten).  The  two  groups  of  o{>erators  showed 
no  significant  differences  in  their  performance.  Their  error  rate  for  select¬ 
ing  the  proper  format  was  22%,  suggesting  that  approximately  one  message  in 
five  would  be  incorrectly  entered  in  a  real  life  situation.  Error  rates  dif¬ 
fered  as  a  function  of  message  type  and  subject  matter.  The  most  common  error 
made  involved  data  being  sent  through  TOS  as  a  relay  message  when,  in  fact,  the 
data  should  have  been  entered  as  an  operational  message.  This  type  of  error 
deprives  the  command  center  of  important  information.  The  mean  time  for  data 
entry  was  observed  to  be  50  seconds.  This  included  the  time  required  to  read 
the  message,  select  a  format  and  then  transform  the  message.  As  a  result  of 
their  findings,  Baker  et  al.  hypothesized  that  the  error  rate  in  an  operational 
system  is  compounded  by  the  large  number  of  successive  operations  being  per¬ 
formed.  They  suggested  that  the  action  officer,  who  manually  prepares  the  mes¬ 
sage  worksheet  and  then  passes  it  to  the  operator  for  input,  is  both  redundant 
and  a  source  of  error.  In  addition,  they  recommend  that  the  number  of  differ¬ 
ent  formats  used  by  the  TOS  be  reduced  and  that  operator  training  on  correct 
choice  of  a  format  be  improved. 
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ARI  Laboratory  Experiments  on  Data  Entry 
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Table  1  (Continued) 


FRIENDLY  UNIT  INFORMATION 


Lgure  4.  Sample  unit  status  query  worksheet.  (Baker  et  al.,  1969) 


•  The  Redundancy  of  the  Action  Officei  ' and  the  Operator  Tod:;,  a:;  d<  - 
l  ined  by  baker  ot  al.  (ltH-d),  was  investigated  ly  ptruh  (i“7l).  jj,  a).  (v.  ,  ri. 

t  designed  tu  exp-lore  alternative  iiumii;;  loj  i  mi  n  .vinij  the  entrv  :  tc-ces;  ,  •  v.i 
mode.-;  ot  format  preparation  (on-line  and  ot  f  -]  ine)  wvti  compared  uiidei  two  i,v- 
-‘1:;  ot  veritication  (witji  and  without).  On-lir.o  entry  used  tin-  t  erif.ji.ul  :  oi 
i  reparation  (see  Figure  5);  off-line  entry  required  the  £  le;  al  alien  ot  a  j  aj  >  : 
t  ormut  p-r  ior  to  data  entry  into  the  terminal  .  1 1.  the  verified  c.-ndit  iot.,  two 

oj  orators  eompared  their  completed  lor  mat  s,  r»  solved  anv  <lisci  e:  at.ei*. ;  1  ot  we<.  • 

the  two,  and  entered  the  formats  individually.  The  unverified  condition  otr.i  tie* 
these  slot  s,  i .  e .  ,  the  oj  orator  performed  the  input  ot  elation  witi.out  at.  -  rr-.r 
cheek.  A  fifth  group  of  operators  served  as.  a  control  .  These  o£  .-ratin'!-  :n- 
1  e-»  tot  mat;,  using  i 'rooeduia.-.-;  similar  to  those  etn[  loyed  in  the  -  xp  r  irtental 

luy,  i.o.,  lot  mats  were  pi  spared  on  £  apet  by  one  oj  eratct  and  t  rar.sct  i  1  od  onto 
the  terminal  by  another  operator.  (in  tho  piot  conditions,  loth  t.:.<  j  r>  j  at  u- 
f  it'll  and  entry  were  i  erfornied  by  a  single  operator.)  S  i-;t.ii  icar;  ly  ,i  ,  ;  t  o„- .• 
wcre  obtained  in  the  on-line  preparation  (11..’  )  than  in  the  ot i -line  :  r-.-j  aru- 
trun  (14. bn).  There  were  no  reliable  differences  between  these  two  condition? 
in  s£  cod  of  input.  L se  of  the  verification  procedure  reduced  errors  a: £  rex i - 
mutely  one- third  (15.7;.  vs  10.J.)  but  required  upj  roximat  i  ly  one- third  more 
time  ( 4 .  S)d  vs  (-.81  min),  both  j  roeodures  were  superior  to  the  inputtin-:  proc¬ 
ess  user!  i.ti  tile  experimental  TOS.  Prom  these  findings,  ft  rub  recommended  oi.- 
line  inputting  to  reduce  errors  and  noted  that  although  the  verification  £ roeo— 
dure  reduced  errors,  the  trade-off  in  time  and  manpower  must  be  considered. 


•  Aids  to  Assist  in  Message  Formatting  were  evaluated  in  an  experiment  by 
Strub  (1975).  Both  of  the  aids  that  were  studied  provided  the  user  with  data 
entry  information,  instructions,  and  examples  of  legal  entries.  One  aid  was 
computer-assisted  and  presented  this  information  on  the  CRT.  The  other  aid  was 
a  checklist  for  formatting  free-text  information  into  computer  acceptable  ter¬ 
minology.  This  aid  was  in  the  form  of  a  handbook.  A  second  variable  that  was 
investigated  involved  the  completeness  of  the  data  entry  format  shown  to  the 
subject.  In  one  condition,  the  subject  indicated  which  subsection  of  the  for¬ 
mat  was  to  be  displayed  on  the  screen.  In  the  other  condition,  the  entire  for¬ 
mat,  including  irrelevant  sections,  was  displayed  to  the  subject.  In  both 
cases,  the  subject  was  required  to  insert  the  appropriate  data  into  the  blank 
spaces  within  the  format.  A  fifth  condition  served  as  a  control.  In  this  con¬ 
dition,  the  subject  used  the  reference  handbook  but  wrote  the  input  message 
without  the  aid  of  a  CRT  display  format.  The  experiment  found  that  the  time 
required  to  perform  the  data  entry  task  did  not  differ  significantly  among  the 
four  experimental  conditions  but  the  fifth  (control)  condition  did  take  an  av¬ 
erage  of  4.6  minutes  longer  per  message.  No  significant  differences  in  accu¬ 
racy  appeared.  An  analysis  of  the  input  errors  was  performed  to  evaluate  the 
effectiveness  of  the  error  detection  routine.  The,  analysis  revealed  that  80t 
of  the  input  errors  went  undetected.  Also,  "friendly"  messages  were  completed 
with  greater  accuracy  and  speed  than  "enemy”  messages,  perhaps  because  they  may 
have  been  simpler  to  format. 

Strub  recommended  that  error  analyses  be  performed  to  determine  the  ratio 
of  detectable  to  undetectable  errors.  This  would  provide  a  valuable  estimate 
of  the  frequency  of  undetected  errors  in  the  real  world.  Strub  also  suggested 
that  the  TOS  formats  might  be  simplified.  Research  performed  in  conjunction 
with  Project  MASSTER  (which  was  to  later  become  an  element  of  the  Training  and 
Doctrine  Command)  indicated  that  the  consolidation  of  formats  could  result  in  a 
savings  of  49  seconds  in  the  time  required  to  enter  a  message  (Strub,  1975) . 
Error  rates  for  the  consolidated  formats  were  not  reported. 

•  Improved  Training  represents  another  approach  to  improved  operator  per¬ 
formance.  Sof tware/courseware  training  modules  can  be  embedded  within  operat¬ 
ing  system  both  to  train  operators  and  to  maintain  a  high  level  of  proficiency 
one i  trained.  The  effectiveness  of  embedded  training  programs  depends  on  the 
development  of  optimized  training  strategies  which  are  responsive  to  the  stu¬ 
dent's  history,  current  skill  level,  etc.,  and  satisfy  the  necessary  conditions 
for  learning  (e.g.,  feedback  of  errors).  An  experiment  by  Gade,  Fields  and 
Alderman  (1978)  compared  the  effects  of  selective  feedback  on  performance  dur¬ 
ing  training  and  simulated  operations.  Five  conditions  were  defined  by  the 
type  of  computer  feedback  provided  to  tiie  student  when  an  error  occurs.  The 
five  types  of  feedback  were: 

1 .  No  Feedback . 

2.  Minimum  Feedback — error  message  only. 

3.  Edit  Feedback--error  message,  correction  required. 

4.  Remedial  Feedback--crror  message,  correction  required,  correct  entry. 

5.  Response  Sensitive  Feedback — error  message,  correction  required,  cor¬ 
rect  entry,  automatic  entry. 
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The  error  message  informs  the  student  that  the  last  entry  was  in  error;  correc¬ 
tion  required  indicates  that  erroneous  entries  must  be  corrected  before  pro¬ 
ceeding;  correct  entry  provides  the  student  with  both  the  erroneous  entry  and 
what  the  entry  should  have  been;  for  automatic  entry,  the  computer  maintains  a 
record  of  the  subject's  performance  and  stops  testing  those  entries  which  have 
been  mastered  to  criteria.  After  training,  the  subjects  were  given  a  perfor¬ 
mance  test.  In  comparison  to  the  other  conditions,  Response-Sensitive  feedback 
was  found  to  be  most  effective  in  reducing  training  time.  However,  it  was  no 
different  in  its  effect  on  accuracy.  Of  the  errors  made  during  testing,  over 
90%  could  not  be  detected  by  the  system's  edit  routines. 

Gade  et  al.  recommended  that  any  proposed  use  of  the  above  training  tech¬ 
niques  should  take  into  consideration  the  trade-off  between  reduced  training 
time  and  increased  cost  of  training  development.  Also,  the  development  of  more 
effective  error  detecting  routines  was  recommended,  although  this  is  not  likely 
to  produce  a  complete  solution.  Other  possible  means  for  reducing  error  rates 
are  to  improve  formats  and  to  better  define  legal  entries.  Based  upon  the  above 
experimental  results,  it  does  not  appear  that  improved  training  will  eliminate 
the  number  of  errors  that  go  undetected  by  the  present  system. 

•  Typing  Aids  were  investigated  by  Fields,  Maisano,  and  Marshall  (1978) 
in  an  extension  of  earlier  work  developing  computer  aids  for  message  input. 

The  authors  systematically  assessed  four  methods  of  inputting  TOS  messages  con¬ 
cerning  enemy  activity.  The  four  inputting  methods  were:  (1)  typing--the  user 
types  the  appropriate  code  into  a  message  format;  (2)  typing  with  an  error 
corrector — the  computer  automatically  attempts  to  correct  common  spelling  and/or 
typing  errors;  (3)  menus — from  a  list  of  options,  the  user  indicates  which  entry 
is  desired;  and  (4)  typing  with  autocorrection  and  an  English  option — the  user 
could  choose  to  type  either  the  word  or  its  abbreviation  (i.e.,  English  option) 
or  the  user  could  choose  to  only  type  enough  characters  from  either  the  word  or 
its  abbreviation  for  the  computer  to  uniquely  identify  it  (i.e.,  autocompletion) 

The  menu  method  of  data  entry  produced  40%  fewer  errors  than  the  typing 
only  method.  The  error  corrector  also  decreased  the  number  of  errors  by  11%. 
However,  autocompletion  with  an  English  option  resulted  in  an  increase  in  the 
error  rate  as  compared  to  the  typing  only  method.  (Although  autocompletion  was 
utilized  heavily  by  the  subjects,  the  English  option  was  not.  Instead,  subjects 
showed  a  strong  preference  for  using  abbreviations  in  conjunction  with  auto¬ 
completion  and  rarely  did  they  use  the  other  options.)  Autocompletion  should 
therefore  not  be  used  unless  specific  operational  needs  warrant  it.  It  also 
might  prove  to  be  more  effective  with  operators  experienced  in  its  use.  For 
users  with  limited  (one  day)  experience,  there  were  no  significant  differences 
in  speed  among  the  four  methods.  The  menu  method  required  a  slightly  (although 
not  significantly)  longer  data  entry  time.  This  may  have  resulted  from  a  lag 
in  the  time  required  to  display  successive  menus  on  the  CRT. 

•  The  Memorability  of  Codes  is  a  problem  for  most  systems.  Early  ver¬ 
sions  of  the  experimental  TOS  used  reference  codes  consisting  of  two  letters 
and  one  number  (LL#) .  The  two  letters  were  used  to  indicate  message  format 
(e.g.,  enemy  unit  status,  standing  request  for  information  file)  and  the  number 
was  used  to  indicate  action  code  (e.g.,  add,  delete).  An  alternative  set  of 
codes  based  on  four  letters  (LLLL) ,  usually  an  acronym  of  the  message  format 
and  the  action  code,  was  recommended.  A  sample  of  the  two  coding  techniuues  is 
shown  in  Table  2.  A  code  is  formed  by  joining  a  message  title  code  to  an  action 
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Table  2 


Old  (LL#)  and  New  (LLLL)  Message  Format  Codes 
Used  in  the  Experimental  TOS 
(Nystrom  and  Gividen,  1978) 


Message  Titles  with  Old  (LL#)  and  New  TLLLL)  Codes 


Message  title  code 

G2  message  titles  Old  New 


Enemy  unit  status  EA  EUS 
Enemy  situation  data  EC  ESD 
(Enemy)  intelligence  summary  ED  EIN 
(Enemy)  intelligence  working  file  EE  EWF 
Enemy  situation  data  base  index  EG  EDX 
Relay3  AA  REL 
Named  area  of  interest  AA  NAI 
Standing  request  for  information  file  AA  SRI 


Action  Names  with  Old  and  New  Codes 


Action  code 

Action  name  Old3  New 


Add  1  A 
Change  2  C 
Delete  3  D 
Query  4  Q 
Special  processing  5  P 
Establish  standing  request  for  information  6  S 


apius  these  special  cases: 

1.  Use  an  action  code  of  0  to  add  a  Relay  message. 

2.  Use  an  action  code  of  8  to  delete  an  "SRI  File"  message. 

3.  Use  an  action  code  of  9  to  query  the  "SRI  File." 


code  (e.g.,  EA2,  EUSA) .  Nystrom  and  Gividen  (1978)  compared  the  relative  ease 
of  learning  the  two  sets  of  message  format-action  codes.  Comparison  of  the 
error  rates  showed  a  reliable  difference  in  learning  the  codes,  i.e.,  the  error 
rate  for  learning  the  LLLL  codes  was  less  than  half  that  for  learning  the  LL# 
codes  (13%  vs  29%  errors  for  enlisted  men,  11%  vs  21%  for  officers  working  on 
one  sample  list,  and  7%  vs  15  for  officers  working  on  another  sample  list). 

In  addition,  the  time  required  to  meet  a  learning  criteria  for  the  LLLL  groups 
was  approximately  60%  of  the  time  required  for  the  LL#  groups.  Thus,  the  LLLL 
code  was  learned  in  less  time  and  with  fewer  errors  than  the  LL#  code.  A  de¬ 
tailed  analysis  of  the  error  rates  suggested  several  revisions  in  the  LLLL  code 
that  have  the  potential  of  reducing  the  error  rate  to  5%  or  less. 

•  Improved  Keyboard  Designs  were  studied  by  Sidorsky  (1974)  in  research 
designed  to  evaluate  the  impact  of  a  nonstandard  keyboard  on  the  data  entry 
process.  Sidorsky  specifically  compared  data  entry  using  his  own  "Alpha-dot" 
five-key  keyboard  versus  a  CRT  with  a  standard  keyboard.  The  Alpha-dot  system 
is  one  in  which  the  keys  to  be  pressed  are  determined  by  the  shape  of  the  al¬ 
phanumeric  character  being  entered.  The  set  of  keys  that  most  closely  resembles 
the  character  is  pressed.  The  Alpha-dot  code  and  two  Alpha-dot  entry  devices 
are  shown  in  Figure  6.  With  less  than  5  hours  of  practice,  trainees  using  the 
Alpha-dot  tablet  were  able  to  enter  free  messages  at  60®  of  their  standard  key¬ 
board  rate.  Perhaps  more  significant  was  the  finding  that  formatted  (TOS  type) 
messages  were  transmitted  at  a  rate  equal  to  or  exceeding  that  of  the  standard 
keyboard.  Subsequent  studies  have  similarly  demonstrated  the  feasibility  of 
the  Alpha-dot  keyboard.  Sidorsky  (1979)  showed  that  enlisted  personnel  were 
able  to  effectively  master  the  system  with  1  to  3  hours  of  practice.  Similarly, 
Gopher  and  Eilam  (1979)  taught  the  Alpha-code  (in  Hebrew)  to  fighter  pilots  who 
mastered  it  quickly  and  went  on  to  successfully  operate  it  during  a  simulated 
flight  mission. 


ANALYSIS  OF  THE  DATA  ENTRY  PROCESS 

In  addition  to  both  field  evaluations  and  labora  ■ '^ry  exp<"  iments,  a  number 
of  analytical  efforts  have  been  performed  in  the  a-'  o  •,  f  data  ,;-'ry.  These  ef¬ 
forts  have  produced  practical  tools  for  aiding  in  t  _■  design  of  data  entry  pro¬ 
cedures.  Two  of  these  efforts  have  provided  the  analytic  tools  necessary  for 
discovering  and  evaluating  error  prevention  techniques.  The  other  efforts  cre¬ 
ated  a  quantitative  model  of  the  data  entry  process.  This  model  can  be  used  to 
inexpensively  simulate,  and  thus  evaluate,  alternative  designs  for  the  data 
entry  process  in  the  TOS. 


Error  Analysis 


Nawrocki,  Strub,  and  Cecil  (1973)  developed  an  error  analysis  technique 
based  on  comparing  the  entered  messages  with  perfectly  accurate  messages.  Any 
mismatches  or  discrepancies  between  corresponding  entries  were  defined  as  er¬ 
rors.  The  errors  were  then  categorized  according  to  the  type  of  failure  that 
occurred  in  the  inputting  process  (e.g.,  omission,  commission).  This  technique 
for  error  analyses  was  first  applied  to  the  data  obtained  in  two  experiments 
performed  by  Strub  (1971,  1975).  Examination  of  these  data  revealed  that  three 
error  categories--omission  of  an  appropriate  entry,  commission  of  an  inappro¬ 
priate  entry,  and  glossary  (i.e.,  "selecting  an  inappropriate  entry  from  among 
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ALPHA-DOT  TABLET  CODE 


a  glossary  of  potential  entries  available  to  the  operator") — accounted  for  about 
80%  of  the  errors.  Yet  these  three  categories  represent  errors  which  cannot  be 
detected  by  computer  edit-and-validate  routines.  Also,  error  rates  were  found 
to  vary  as  a  function  of  the  type  of  message  being  processed.  "Enemy  activity 
messages  produce  a  higher  error  rate  than  friendly  messages  and  the  bulk  of 
this  difference  is  in  the  category  of  omissions."  The  need  to  reduce  this 
high,  undetectable  error  rate  has  provided  the  impetus  for  work  on  identifying 
both  the  sources  of  errors  and  how  errors  can  be  countered. 

The  prevention  and  remediation  of  input  errors  was  approached  by  Mace, 
Harrison,  and  Sequin  (1979).  Through  the  creation  of  a  taxonomy,  these  authors 
sought  to  identify  ways  which  input  errors  are  made  as  well  as  ways  by  which 
they  can  be  prevented.  The  report  consists  of  four  basic  components:  a  scheme 
for  error  classification,  a  determination  of  the  constraints  imposed  upon  error 
remediation,  an  investigation  of  the  design  changes  which  might  result  in  a  re¬ 
duction  of  errors,  and  a  methodology  for  assessing  the  cost-effectiveness  of 
alternative  remedial  actions.  Each  of  these  components  are  discussed  later.  A 
sample  analysis  is  shown  in  Table  3.  Mace  et  al.  categorize  input  errors  ac¬ 
cording  to  a  number  of  dichotomous  variables.  For  example,  an  input  error  can 
be  one  of  either  omission  or  commission;  it  can  either  be  caught  by  the  internal 
edit  program  or  it  cannot;  it  can  involve  either  a  quantitative  or  a  verbal 
statement.  By  first  explicitly  identifying  the  nature  of  input  errors,  the 
authors  can  go  to  discuss  means  by  which  their  numbers  can  be  reduced. 

Before  viable  remedial  steps  are  devised,  the  system  constraints  must  be 
determined.  All  corrective  actions  are  not  necessarily  feasible.  The  task  and 
design  of  the  system  place  limits  on  what  changes  can  be  made.  Two  facets  of 
system  constraint  are  considered:  (a)  external  influences  such  as  the  unit's 
size  and  weight  limitations;  mobility;  temperature,  dust,  and  humidity;  sup- 
portability,  reliability,  and  maintainability,  and  (b)  internal  influences  con¬ 
cerned  with  distinctly  human  factors  (e.g.,  operator  ability,  training,  and 
experience) . 

Having  determined  the  types  of  errors  and  the  system  constraints,  alterna¬ 
tive  means  by  which  to  reduce  the  number  of  errors  can  be  considered.  Possible 
design  changes  are  revised  formats,  improved  abbreviations,  changes  in  the  input 
language,  display  of  the  default  values,  etc.  Mace  et  al .  caution  against  over¬ 
zealousness  in  instituting  a  change  for  the  purposes  of  eliminating  errors. 
"Elaborate  error  prevention  and  detection  procedures  may  have  a  negative  impact 
on  system  responsiveness  which  may  in  turn  negatively  impact  on  user  acceptance 
and  the  production  of  errors  from  boredom  or  lack  of  confidence  with  the 
system. " 

Having  settled  on  a  possible  preventive  or  remedial  action,  a  system  de¬ 
signer  must  analyze  the  prospective  costs  and  benefits  involved  in  its  imple¬ 
mentation.  Mace  et  al.  provides  a  step-by-step  description  on  how  to  perform 
this  analysis  using  Multi-Attribute  Utility  Measurement  (MAUM) .  The  end  prod¬ 
uct  of  the  procedure  is  a  utility  value.  In  deciding  between  a  number  of  al¬ 
ternative  corrective  actions,  the  system  designer  will  choose  the  one  which 
results  in  the  largest  utility  value. 
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Error  Types,  Causes,  and  Alternatives  for  Prevention  and  Detection 

(Mace  et  al . ,  1979) 


transpos ition  and  decimal 


Simulation 


The  development  and  testing  of  a  new  system  is  an  expensive  endeavor.  In 
order  to  reduce  the  costs  involved  in  evaluating  alternative  designs  and  proce¬ 
dures,  MANMODEL,  a  computer  simulation  of  the  data  entry  process  was  developed 
with  ARI  support.  MANMODEL  focuses  on  system  tasks  and  procedures  in  an  effort 
to  estimate  performance  as  it  pertains  to  data  entry.  By  varying  input  param¬ 
eters,  MANMODEL  permits  system  designers  to  explore  the  effects  on  the  data 
entry  process  of  changes  in  manning  levels,  training,  personnel  selection  and 
other  factors.  Thus,  MANMODEL  helps  identify  problem  areas  and  provides  a 
technique  for  evaluating  potential  solutions  in  a  simulated  context. 

MANMODEL  is  based  upon  a  qualitative  model  of  system  performance  developed 
by  Baker  (1972b,  1976)  and  Baker  and  Ringel  (1968) . 

The  model  has  been  developed  along  three  basic  dimensions  which 
are  reflected  in  Figure  7.  The  first,  that  of  data  flow  and  the 
processing  it  requires,  constitutes  one  dimension  of  importance. 

Hence,  it  is  critical  to  "flow  chart"  the  sequence  of  events  or  oper¬ 
ations  that  constitute  the  logic  of  the  system  under  examination.  .  .  . 

For  the  second  dimension.  Task  Analysis,  each  event  in  the  data 
flow  sequence  is  examined  with  respect  to  the  task-equipment  inter¬ 
actions  that  constitute  that  portion  of  the  operator's  job.  .  .  . 
Essentially,  a  task  analysis  consists  of  the  enumeration  of  the  dis¬ 
criminations,  decisions,  and  action  responses  which  are  necessary  and 
sufficient  to  operate  each  component  of  the  system. 

The  third  dimension  refers  to  outside  sources  of  variation  which 
are  normally  considered  external  to  the  man-computer  system,  for  ex¬ 
ample,  the  impact  on  system  performance  when  level  of  training  is 
varied.  .  .  .  (Baker,  1976,  pp.  7-8) . 

A  MANMODEL  run  begins  with  the  generation  of  a  queue  of  incoming  messages. 
Through  a  stochastic  process,  the  speed  and  accuracy  with  which  those  messages 
are  processed  is  simulated.  Along  the  way,  numerous  variables  relating  to  op¬ 
erator  performance  (e.g.,  stress,  fatigue)  are  calculated.  (A  sample  listing 
of  both  the  input  and  output  variable  used  by  MAN.IODLL  are  shown  in  Table  4.) 

At  various  points  in  the  simulation,  output  reports  are  available  to  describe 
the  status  of  the  data  entry  process.  These  reports  include  such  information 
as  to  how  backed  up  the  message  queue  is,  what  percentage  of  the  time  the  oper¬ 
ators  are  busy,  etc.,  (see  Table  4).  A  small  sample  of  the  simulated  data  that 
is  generated  by  MANMODEL  is  shown  in  Figure'  8.  The  simulated  data  shows  the 
number  of  messages  that  have  been  received,  backlogged  and  either  completed, 
rejected,  or  interrupted  by  two  types  of  data  entry  personnel  (G3  action  offi¬ 
cers  and  IOD  operators).  Further  insight  into  MANMODEL  can  be  gained  by  inspec¬ 
tion  of  the  sensitivity  tests  perform-  d  on  it.  Figure  9  shows  the  predicted 
relationship  between  type  of  operator,  level  of  operator  skill  and  mean  number 
of  messages  which  are  completed  as  opposed  to  messages  that  are  carried  over 
from  one  shift  to  another.  A  complete  description  of  how  MANMODEL  operates  and 
the  data  that  it  produces  can  be  found  in  the  following  documents:  Siegel, 

Wolf,  and  Leahy  (1977);  Siegel,  Wolf,  Leahy,  and  Ftearde  (1977);  Leahy,  Lautman, 
Bearde,  and  Siegel  (1977);  and  Leahy,  Siegel,  and  Wolf  (1979a,  b) . 
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three  basic  dimensions  are:  data  flow,  task  analysis, 
sources  of  variation.  (Baker,  1972b) 


Table  4 


Sample  Input  Parameters  and  Output  Options 
Available  with  MANMODEL 


Input  parameters 


Output  options 


Simulation:  Run  title,  number  of 

iterations,  output  print  options. 

Systems:  Personnel  per  duty  posi¬ 

tion,  shift  duration,  correlation 
and  relative  weights  of  effective¬ 
ness  components,  etc. 

Function/Tasks:  Tasks/duty  level, 
criticality,  segment  delimiters, 
branching,  performance  time  and 
errors,  etc. 

Personnel  Characteristics:  Speed, 
precision,  stress  threshold,  level 
of  aspiration. 

Message  Characteristics:  Number  of 
messages,  message  length,  frequency 
by  type  and  priority,  error  rate  by 
type ,  etc . 

Error  Message:  Vocalic  center 
groups,  search  options,  random  walk, 
etc . 

Interruption/Transmission:  Time, 
hour,  frequency,  etc. 

Computer  System  Characteristics: 
System  delay  time/queue,  number  of 
entries,  etc. 


Effectiveness  Run  Summary:  Thor¬ 
oughness,  responsiveness,  complete¬ 
ness,  and  accuracy  with  overall 
efficiency  for  each  hour. 

Manpower  Utilization:  Proportion  of 
hour  worked,  messages  processed  and 
mean  time,  final  stress  and  aspira¬ 
tion,  and  overall  mean  for  each  hour 

Workload  Summary:  Message  backlog 
and  messages  delivered,  completed, 
rejected,  and  interrupted  by  person¬ 
nel  type  for  each  hour. 

Time  Segment  Summary:  Time  and  pro¬ 
portion  of  total  for  each  segment, 
message  type  and  priority  over  each 
hour . 

Error  Message  Processing:  Mean  time 
spent  responding  to  and  processing 
error  messages  by  each  personnel 
type . 

Error  Run  Summary:  Number  of  error 
returns,  information  loss  and  number 
of  message  units  for  each  hour  and 
message  type. 


DETAIL  PRINTOUT  F()«  INSPECTION  3-19-72  APS 
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Figure  8.  Sample  run  summary  from  MANMODEL. 

(Siegel,  Wolf#  and  Leahy,  1977) 
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Figure  9.  Sample  sensitivity  test  performed  on  MANMODEL. 

Mean  number  of  messages  carried  over  and  completed 
as  a  function  of  operator  skill  level. 

(Siegel,  Wolf,  &  Leahy,  1977) 
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MEAN  MESSAGES  COMPLETED 


For  the  system  designer,  MANMODEL  offers  a  low  cost  tool  with  which  to  ex¬ 
plore  a  large  variety  of  design  alternatives.  By  way  of  illustration,  consider 
a  situation  where  a  proposed  system  design  produces  ineffective  performance 
during  a  simulated  run.  Examination  of  the  simulation  data  might  indicate  that 
it  is  the  system's  responsiveness  (as  opposed  to  thoroughness,  completeness,  or 
accuracy)  which  is  responsible  for  the  poor  performance.  The  average  message 
handling  time  and  the  average  queue  time  are  both  excessive.  A  continued  exam¬ 
ination  of  the  simulation  data  might  also  show  that  personnel  are  busy  a  high 
percentage  of  the  time.  The  system  designer  can  now  examine  the  remainder  of 
the  data  in  an  effort  to  determine  the  probable  cause  of  these  events.  For  ex¬ 
ample,  an  inspection  of  the  Error  Run  Summary  might  indicate  that  the  system's 
error  correction  procedures  are  a  drag  on  the  system.  To  remedy  this,  the  sys¬ 
tem  designer  might  consider  a  variety  of  corrective  measures  such  as:  improved 
operator  training  (Gade  et  al.,  1978);  improving  the  input  procedures  (Fields 
et  al.,  1978;  Sidorsky,  1974);  modifying  the  procedures  so  as  to  reduce  the 
number  of  undetected  errors  (Strub,  1975) ;  or  reducing  the  cognitive  load 
(Nystrum  &  Gividen,  1978) .  Each  of  these  candidate  solutions  may  be  simulated 
on  MANMODEL  and  the  anticipated  performance  benefits  compared  on  the  basis  of 
overall  system  requirements  and  cost  of  implementation. 

MANMODEL  was  initially  developed  for  use  on  a  large  scale  computer  with 
batch  processing.  However,  the  program  has  since  been  modified  to  allow  on¬ 
line  manipulation  of  input  parameters  as  well  as  control  over  program  execu¬ 
tion.  Additional  improvements  and  enhancements  that  have  been  made  to  MANMODEL 
include  the  simulation  of  the  error  correction  process  and  the  acceptance  by 
MANMODEL  of  data  provided  by  a  prior  simulation  of  the  communication  network 
(Leahy  et  al.,  1979a,  b) .  In  the  latter  case,  the  human  performance  predictions 
made  by  MANMODEL  are  then  used  as  input  parameters  for  a  simulation  of  a  tacti¬ 
cal  operations  system.  A  computer-experimenter-subject  interactive  mode  has 
also  been  developed  (Leahy,  Lautman,  Bearde,  and  Siegel  (1977);  Siegel,  Wolf, 
Leahy,  and  Bearde  (1977)).  This  permits  the  simulation  to  make  use  of  actual, 
real-time  data  provided  by  a  subject-operator  who  is  interacting  with  the  sys¬ 
tem.  In  this  mode,  subjects  perform  system  tasks  and  their  data  are  reduced 
and  treated  statistically  for  use  in  the  simulation. 

In  summary,  MANMODEL  provides  an  evaluative  vehicle  for  comparison  of  al¬ 
ternative  system  configurations,  operational  procedures,  and  personnel  charac¬ 
teristics  (e.g.,  training,  aptitude,  motivation).  It  also  permits  the  system 
designer  to  identify  and  manipulate  critical  human  factor  parameters,  to  empir¬ 
ically  investigate  design  alternatives,  and  to  determine  critical  issues. 


GENERAL  DISCUSSION 

Research  on  the  data  entry  process  has  been  responsible  for  refining  the 
input  procedures  and  improving  the  transform  operation  in  battlefield  automated 
systems.  Of  the  seven  experimental  studies  cited  in  this  paper,  the  recommen¬ 
dations  of  three  have  already  been  implemented  and  the  recommendations  of  two 
others  are  currently  being  considered.  However,  in  spite  of  considerable  prog¬ 
ress,  the  basic  parameters  of  the  data  entry  process  have  not  been  fully  defined 
or  investigated.  One  reason  for  this  shortcoming  is  the  lack  of  compatibility 
in  the  message  formats  (stimuli) ,  personnel  and  other  factors  used  in  the  vari¬ 
ous  experiments  reported  above.  This  lac);  of  compatibility  between  experiments 
makes  it  difficult  both  to  reconcile  conflicting  findings  and  to  formulate  a 


total  theory  from  the  individual  findings.  Some  of  the  factors  which  have  been 
found  to  vary  greatly  from  experiment  to  experiment,  and  which  thus  led  to  this 
situation,  are  listed  below. 

1.  Measurement  of  input  times — No  apparent  consensus  has  been  reached  as 
to  how  to  measure  data  entry  times.  Some  experiments  report  the  time  required 
to  input  an  entire  message  while  others  report  only  the  time  required  to  per¬ 
form  discrete  parts  of  the  data  entry.  The  time  measures  used  are  specific  to 
the  experimental  objectives  and  depend  upon  task  characteristics.  They  are  not 
amenable  to  cross  comparison  or  even  to  a  chronometric  approach  of  additive 
task  times  without  caveats  for  cautious  interpretation. 

2.  Stimulus  set — Depending  upon  the  message  formats  (e.g.,  to  make  a 
change  in  the  data  base  vs.  query  the  data  base)  and  message  type  (e.g., 
"friendly"  vs.  "enemy"  messages)  used  as  stimuli  in  an  experiment,  different 
error  rates  will  be  observed.  This  makes  it  difficult  to  compare  experiments 
that  use  different  classes  of  stimuli. 

3.  Instruction  set — Although  this  variable  is  difficult  to  assess  because 
it  is  not  reported  in  detail  in  all  studies,  it  is  obvious  that  some  research 
(either  implicitly  or  explicitly)  emphasizes  accuracy  over  speed  while  other 
research  gives  equivalent  weights  to  the  two.  None  of  the  experiments  appear 
to  emphasize  speed  over  accuracy.  The  results  of  many  of  the  experiments  re¬ 
ported  above  may  have  been  affected  by  the  ability  of  operators  to  trade-off 
between  speed  and  accuracy,  giving  emphasis  to  one  or  the  other  (Howell  & 
Kriedler,  1963).  On  a  related  matter.  Baker  (1963)  recommends  the  use  of  pro¬ 
grammed-  instruction  techniques  so  as  to  eliminate  "delivery  of  instructions"  as 
a  variable  within  an  experiment  and  also  to  eliminate  experimenter  bias  in  the 
delivery  of  instructions. 

4.  Effect  of  practice  on  operator  performance--Evidenco  indicates  that 
the  performance  of  data  entry  tasks  shows  improvement  with  practice  over  periods 
at  least  two  years  long  (Klemmer  &  Lockhead,  1960,  1962a,  1962b) .  In  addition, 
the  actual  mode  of  operation  and  the  abilities  used  in  entering  data  change 
with  practice  and  with  performance  improvement  (Fleishman,  1960,  1965;  Leonard 

&  Newman,  1964).  Well  skilled  data  entry  operators  use  the  redundancies  in  the 
input  data  to  encode  those  data  into  "higher  order  units"  which  are  then  proc¬ 
essed  as  wholes;  on  the  other  hand,  unskilled  operators  use  a  character-by¬ 
character  mode  of  entry  (Seibel,  1972).  For  some  research,  the  nature  and  ex¬ 
tent  of  the  practice  and  the  training  of  the  subjects  was  not  specified.  Thus, 
the  data  entry  rates  (especially  error  rates)  that  are  reported  may  have  been 
differentially  influenced  by  the  amount  of  practice  provided  to  the  operator 
subjects . 

5.  Variability  of  subject  (operator)  samples — The  range  of  operator  fa¬ 
miliarity  with  and  expertise  in  the  operation  of  the  TOS  system  was  consider¬ 
able  in  the  different  exper iments--from  officers  working  on  TOS  development  and 
knowledgeable  in  G3  operations  to  enlisted  personnel  with  low  GT  scores  and  no 
prior  experience  using  TOS  procedures.  This  variability  may  have  contributed 
to  tlie  differences  in  error  rates  and  data  entry  times. 
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6.  Tho  type  of  input  device — Extensive  reviews  (Alden,  Daniels  &  Kanarick, 
1972:  Seibel,  1972)  have  shown  that  error  rates  and  data  input  times  may  vary 
as  a  function  of  the  data  input  device.  Although  the  standard  "QWERTY"  key¬ 
board,  usually  employed  in  a  CRT  terminal,  was  used  most  often,  other  devices 
such  as  a  keyboard  without  a  CRT  or  handwritten  messages  were  also  used  in  some 
experiments. 

A  second  shortcoming  in  the  present  research  was  the  lack  of  focus  on  the 
basic  distinction  between  the  dynamic  cognitive  processes  that  are  involved  in 
a  task  (such  as  retrieving  information  from  memory  and  using  it  to  transform  a 
message)  versus  the  relatively  mechanical  aspects  of  a  task.  Fields  et  al. 
(1978)  acknowledged  this  distinction,  at  least  implicitly,  when  they  noted  that 
their  "menu"  format  method  yields  fewer  errors  on  an  input  task  than  does  vari¬ 
ous  typing  methods.  They  attributed  this  to  the  possibility  that  fewer  errors 
will  occur  because  the  use  of  menus  places  a  smaller  cognitive  load  on  the 
user,  as  it  requires  the  use  of  recognition  memory  rather  than  recall  memory. 
Two  ways  by  which  errors  can  be  made  while  translating  information  into  com¬ 
puter  readable  forms  are:  (1)  the  problem  of  forming  a  correct  concept  of  the 
information  and  (2)  the  problem  of  inputting  properly  formatted  information 
based  on  a  correct  concept.  Attributing  errors^to  one  of  the  two  causes  men¬ 
tioned  above  becomes  difficult  when  the  experimental  paradigm  confounds  cogni¬ 
tive  processes  with  rote  motor  behavior.  If  the  basic  parameters  which  affect 
the  cognitive  versus  motor  aspects  of  the  data  entry  process  are  delineated, 
differential  weightings  (e.g.,  as  a  function  of  task  or  input  device)  may  be 
both  experimentally  and  then  operationally  assigned  to  them.  Isolating  the 
cognitive  versus  motor  aspects  of  the  data  entry  task  and  assessing  the  vari¬ 
ables  which  impact  upon  them  is  a  potentially  fruitful  area  of  research. 

Other  potential  areas  of  research  are  the  linguistics  of  user-computer 
communication,  training  methods,  operator  selection  techniques  and  keyboard 
design.  With  regard  to  this  last  item,  the  current  state  of  the  art  seems  to 
accept  the  QWERTY  keyboard  as  standard  m  spite  of  evidence  to  indicate  that 
other  keyboards  may  just  be  as,  if  not  more  effective  for  certain  types  of 
tasks.  Research  on  the  technical  parameters  of  keyboards  (Alden  et  al . ,  1972) 
and  the  characteristics  of  keyboard  operators  may  lead  to  rapid  improvement  in 
the  data  entry  process.  Finally,  research  should  consider  the  emerging  tech¬ 
nology  for  computer  input  and  output  of  speech  coupled  with  the  apparent  bene¬ 
fits  of  entering  data  at  the  source  (i.e.,  observed  on  the  battlefield). 


SUMMARY  AND  CONCLUSIONS 

A  number  of  ARI  research  studies  have  investigated  techniques  for  improv¬ 
ing  the  TOS  data  entry,  process  through  procedural  and  hardware  changes.  Many 
of  these  studies  report  reductions  in  the  time  required  to  enter  data  and/or 
reductions  in  the  error  rate.  The  degree  to  which  this  research  pirogram  has 
improved  performance  is  shown  in  Figure  10.  The  more  important  findings  and 
operational  implications  that  can  be  derived  from  this  program  are  listed 
below. 
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TOS  DATA  ENTRY  PROCESS 
RESEARCH  HIGHLIGHTS 


REFERENCE  A:  JOB  AID  PRODUCED  APPROXIMATELY  5%  FEWER  ERRORS  THAN 
DID  THE  TOS  MENU). 

.  A  MINUS  SIGN  WITHIN  A  BAR  INDICATES  THAT  THE  IMPOVEMENT  WAS  NEGATIVE 
(E  G.,  TIME  TO  PERFORM  THE  TASK  INCREASED). 


1 .  The  number  of  different  formats  used  in  data  entry  should  be  reduced 
through  possible  consolidation.  Previous  TOS  designs  have  contem¬ 
plated  using  up  to  500  different  formats.  Such  a  large  number  of 
choices  makes  it  difficult  for  the  operator  to  properly  select  the 
appropriate  format.  (This  recommendation  has  been  implemented.) 

2.  The  transformation  and  input  of  the  TOS  message  should  be  performed  by 
a  single  operator  and  as  a  single  operation.  The  intermediate  step  of 
completing  a  paper  format  as  well  as  the  role  of  an  action  officer  are 
redundant,  error  contributing  and  should  therefore  be  eliminated. 

(This  recommendation  has  been  implemented.) 

3.  Menu  selection  is  the  superior  technique  for  data  entry.  However,  one 
must  still  determine  the  most  efficient  form  for  setting  up  a  menu. 
(This  recommendation  is  being  implemented.) 

4 .  Message  formats  are  best  designated  by  the  use  of  all-letter  code  as 
opposed  to  a  mixed  alpha-numeric  code.  (This  recommendation  has  been 
implemented. ) 

5 .  Flexibility  and  cost  efficiency  in  the  design  of  data  entry  systems 
can  be  achieved  through  improved  computerized  modeling.  MANMODEL  rep¬ 
resents  a  case  in  point.  (This  recommendation  has  been  implemented.) 

6.  Response-sensitive  training  (where  the  training  session  automatically 
adjusts  to  changes  in  the  operator's  performance)  is  an  effective 
method  for  reducing  training  time  without  sacrificing  input  accurac; 
However,  the  decision  on  whether  to  implement  such  a  training  tech¬ 
nique  requires  an  analysis  of  its  cost.  (This  recommendation  is  under 
consideration. ) 


The  Alpha-dot  Keyboard  provides  an  alternative  data  entry  device  for 
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